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ABSTRACT 

In this manual, project direc 
Reading Instructional Teams (IRIT) program in 
public schools, are provided with suggestions 
compensatory programs such as' the IRIT.* Three 
I evaluation are discussed 
control group design, and a spec: 

for using the models are given, and suggestions^ for modifying them to 
fit particular needs, jare presented., The format of an evaluation 
report is outlined as including a project description, strengths 
and/or accomplishments, probPem areas , evaluation objectives, 
procedures, and .f indingS and recommendations. The heed for 
dissemination and feedback of evaluation results is stressed. An 
appendix contains copies of Hartford's evaluation reporting forms, 
project data coding sheets, and an IRIT replication reporting form. 
(CMG) 
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" AN IRIT DIRECTOR* S GUIDE TO EFFECTIVE EVALUATION 

Congratulations. - And welcome to the Intensive Reading Instructional 
Teams family. We^feel certain that your youngsters, their teachers , and 
their paints will like the IRIT and will be ple'ased with the educational . . 

• * » ; ' ■ ' • • 

growth which it produces. 

Did you know that the original Hartford IRIT was started in 1965 and 

v is .still going strong? While we won'f dwell on past accomplishments, there 

is an historical point to be made: because the first IRIT program was started 

« 

with Title I and State compensatory, monies , an evaluation was required from 

* 

.* i» . • ■* 

the outset. It was ttjis requirement which enabled the IRIT to qualify for 

*> 

federal recognition as early as 1969. You, too, may be in the same position 
where ^n evaluation is required. ' Even if a,n evaluation is not required by 
your district, it is a gqod idea to evaluate just the same. 

Why should I evaluate? And what's in it for me? Right now,. money 
for ec^cation is tight.' In consequence, taxpayers want to know if a program 
is working and if it is worth the dollars which are being spent. „ . 

Taxpayers also want to look at program results so as to capitalize 
On strengths and correct obvious weaknesses. . , 

Of course, there is always that reporting requirement; sometimes it 
is to a funding agency but often to parents, teachers, the Board of Education, 

* f c 

and the general public as well. , * 

But how do I get a useful evaluation? First, let's loo)£ at the facts. 



- * There is no one correct approach to an evaluation. * 

To be useful, an evaluation responds to the needs of its, .. 

" audiences, to program concerns,, isstfes and objectives . 

i 

No evaluation is perfect. Almost every^evaluation has at 

leas r t some methodological problems..-. -.in the instruments, with . 

the procedures , 3,nd with the application of the results to the 

constraints of the real world. . * \ . - 

Evaluation is not research. While there are many similarities 

■ 

between the two, researchers want to draw general conclusions, 

v ^ < 
while *evaluators want to facilitate specific decisions. 

Let's examine our facts more closely. Most evaluators will tell you 

that an evaluation is a systematic way to assesjpedifSational worth. • They 

will also tell you that the main purpose fpr most evaluations is to help make 

t * * ■■ 

a program better. Evaluations provide information for decision-making by 
the project staff, the administration, the Board of Education / the funding . 
agency and sometimes even the parents.^ * * ^ 

While, no. one evaluation will provide all thjb decision-making, informa 

•» 

tion which is needed by these audiences, jnost evaluatiohs will be aimed at 
one or more target groups.. Since evaluations are usually intended to help 
.staff make program improvements, an evaluation should be geared to this 
audience and their needs considered . m v 

An IRIT evaluation should provide necessary information to your own 



administration and to your 'funding .agenc^T Both audiences want to know 

• i 

howthe money is being spenf,^ how-it can spent more- effectively, and 

' °- ' * * • ' • 

how administrative and management* elements can be improved/ Since both m 

audiences are more concerned With managerial and policy decisions; the 

information needed is less detailed than that required by the project staff. 

Even if the evaluation is designed v to meet the needs of the project staff, it 

should also 'attend to the requirements of these other audiences. 

Thei '^ is no one approach to the evaluation of a- compensatory- program 
• • • , 

such as the IRIT. However, -there are some suggestions. Ah IRIT evaluation^ 

^should deal with the stated program objectives and with the implied objec- 

tives as well. Since the IRIT was dr.signed'to improve reading achievement,. 

v 

appropriate measures of reading achievement are needed. If your IRITBis 
attempting to motivate youngsters to read books on their own time and if 
this area constitutes a major program- thrust, this area should be a focus of 
the evaluation, whether or not an objective has been stated. 

The evaluators 1 creed, if there was one, woukj probably be Murphy's 
Law. . .if things, can go wrong, they will. Murphy's Law will usually con- 



found even the best evaluation. With the best of planning, a perfect 



evaluation is almost impossible/ Even so, the idea is to use the be* 
model, the best instruments, and^the best methodology possible and tailor 

"these to fit into the real world. This is not as difficult as .it xnay at first • 

6 



seem , 



Since the IRIT is a relatively straightforward program (so far as 




evaluations are concerned) , a series 
of usable models can be suggested. 
. For those 1 who wish to delve more 
deeply into the ^evaluation business 
"or'vtould like to know>more about' 

i 

evaluation theory, models, and 
methodology, several references 
have been listed in the appendix. 
The' Real World of Evaluation - 
Before you sit down to work - 
w - out an evaluation plan, there are . 

i * • , -j 

■ r * <* 

several qilestiofis which must be 
answered These Questions attend to vour own situation and to yout Ideal ^ 
school district's ground rules, , ; • 

V .' ~ ; ■. ' . 

What is your district 1 s -policy conceming-eyaluations? Are there * 

! ».- * 

a series of "canned""procedures which must be followed? Or are 
you free to go it alone? . . , 

- HoV is your project funded? WhiJe each outside funding source 
has its own ?et of evaluation requirements , most are quite similar 

• Ydu should be aware of any specifics which apply to your project 
and follow the funding rules . 

- What evaluation resources are Available to you? Is there an 



evaluation .office which will assume full responsibility for the 
total evaluation effort, or will you have only the- resources that 
you can beg, borrow, or steal? For evaluation purposes , 
resources are usually classified , in terms of time, money, facil- 
ities, personnel, and technical expertise. - 
-.. What other kinds of questions' do you need to have answered . 
For example, you will need to identify your audiences and their 
infoimation needs. These questions and answers must be* iden- 
'tified on the. basis of local needs. * . 
Once the applicable ground rules.have been determined, it time 
to-begin thinking about an evaluation approach which will attend to your' 
IRIT audienc^ Let's first discuss several evaluation terms apd concepts. 

The evaluation literature often refers to surpmative ^product) md formative 

• ■> 

(process)- evaluatioris . A. product. or summative evaluation famines out- 
puts derived from measuring the behavioral objectives specified in your < 
program or your funding proposal. A product "evaluation examines instruct 

. r '■ . v . . ' 

tional impact on the program's clients: the pupils, teachers, and parents. 
A product evaluation looks at outputs in relation to the services which 
were provided . , i 

A formative or process evaluation is concerned with how a program 
works and what processes are used to bring about client change. A process 
evaluation is particularly important to'a developmental program since it is 



important for decision makers not only to know what has happened, but • 

how. Since the IRIT is an established exemplary progra.rn, a process 

evaluation will have limited impact upon the program. This is why this 

booklet focuses primarily on evaluating program outcomes. 

Title I, like most federal programs requires an annual evaluatipn, 
i . »■ ' . 'i 

At the" very least, this evaluation must include pre-and post-test scores. .• 

* « • • 

Annual Title I evaluations also require the reporting of various kinds of 

information. . .program costs, the number of staff hired / the number. of 

youngsters served, and other evidences of program effort. 

Title I is the largest federally-funded education program- Its . 

funded services run the £amut of instructional programs, approaches, and 
* § 

target audiences. Because of this size and complexity. Title I prbgrams 
are required to use a rigorous evaluation model.- These models were 
designed to assess the impact of -Title Tfunding nationwide. With few 
exceptions, the models require the use of standardized tests that are 
properly administered. Test scores must be recorded .in a common manner, 
and compatible pre and post test data must be obtained. .ThQ Title. I 
models, were designed to focus only on test data and must be supplemented 
with additional information for local -school district use. The Title I model 
however,, are suitable for use with IRIT programs which are funded from a. 
variety of sources . . . 

r 

* / 



. Title I Evaluation Models 

i * ^ s . . \ 

« ' The basic Title I evaluation' scheme employs one of three models. 

The -models are designed to help answer the question, "How much more did 

the students learn with classroom instructic>n and IRIT services than they 

would have learned with classroom instruction alone/? 11 Test score*gains • 

i 

attributable to the IRIT treatment are defined as the difference bet^eep^the 

IRIT group's performance on a post-treatment test and an estimate of what 

* • t 

the performance on the same test, would have been if the group dicj not ' 
receive the IRIT treatment. The measure of IRIT impact is the obsferved • 

i 

post-treatment performance minus the expected no-treatment performance. 

IRIT impact looks like this: 

• / ■ ' ■ ' 
IRIT IRIT EXPECTED . 

Impact = OBSERVED - " CLASSROOM . ;J 
EFFECT « (-NO IRIT) 

• •„ EFFECT 

The ohserve'd post-treatment performance is always the mean or 
median post^test score of the IRIT treatment group. The no-treatment 
expectation is determined by using one of the three Title I evaluation model 
.Model A: Norm-Referenced Design 

The norm-referenced design assumes jthat, without the special IRIT 
treatment, the pupils as a group wou}d have maintained the same relative , 
status to a local or national norm group from pre- test to post-tjest. TlTe 
group's' average pre-test percentile is not expected I % to change if no special 



treatment's provided The group pre-test percentile is Expected to T^e the 
same as the group's post-test percentile if no IRIT treatment occurred. 

The observed post treatment performance is simply the percentile | 
rank- which corresponds to the group's mean post-test score. If the groups 
post-test percentile is higher than the 'pretest percentile, it may be 
assumed that the improvement resulted from IRIT participation^ . 

. While the Model A norm 7 referenced design can be used with normed 

' * "* - . 

or innorm'ed te'sts, we recommend pre and post testing with the appropriate 

■'[■'■ . j- ' ' , 

form and level of a nationally normed achievement .test. This testing 

* should take place at times whith are as closfe as possible^to the dates 

when the norm data were originally collected. Most tests are normed in 

the spring andjn the fall. Thus, fair to spring; or spring to spring testing 

i . ' ■ 

for each IRIT cycle is recommended. 

Model B:„ Control Group Design . 

The control group, or 

research design model requires^ that 

the' same standardized achievement 

test is given to both an IRIT treat- 

ment group arid a comparable group" ( 

on* a pre and post service basis 1 



The design assumes that both the 
treatment and control groups are 




essentially the same. .If the IRIT treatment group's post-test performance* * 

is superior to that of the contrQl gfoup, the IRITiprogram has been effec- 

- — ..... " « ^ ;> 

tive/ The control group model compares treatment grbup data with the same 

: t ■ ' 

kiticjs of data which have been obtained from a comparable group of young- 
sters in the-same" school or district." The control group design, is a very 

powerful model. It assumes that the only difference between the two 

' ^ • * 7 ■ 

groups will be the IRIT treatment. Thus, a superior IRJT post-test perform- 

ante provides a powerful -argument that the IRIT program ha^ made the . 

*. ■ , !*•■'** 

\ 

difference. I 

The key to Model B "success is to select control and treatment group 

which are comparable. The problem is that the best 'way to equate these * 
■ • " - *. '. \ ■ ■ * 

groups is to randomly assign youngsters to the t r eatment and to the control 

group. In a 'public scho'ol- thi^ procedure is oftetrimpossiBle, impractical, 

or under Title I funding possibly illegal . As a c^n'sequen'ce , most Title I 

programs db nol use. this modej/ despite its rigor ancj power. t 

a For the IRIT program, the Model B picture is much more/rpsy. In 
: ' - f ' .\- 

fact, thefre are^some techniques wlri^h can be used tp eliminate many of the 

' * 1 \ ^ 

problems associated with random -sampling; While th£ control group design 

\ * : * 

may be the model of choice , its implementation requires a rather tyigfy level 
of evaluative expertise and the services ol a computer. If you have these 
resources, the ModeJ B should certainly be considered-. . 



Model C: Special Regression Design * - 

With the regression design, an IRIT treatment group and a superior 

I 

comparison- group are formed from one "intact" group on "the basis of a pre 
test cut-off score. The special regression design projects expected perform- 
ance scores for both the IRIT treatment and the IRIT comparison group using 
a statistical procedure, A selection test is given to all eligible pupils. 
Students falling below the cut-off score are placed in the IRIT treatment 
group, while those scoring above the cutoff are used as the comparison 
group. If the obtained IRIT group score is higher than the statistically 
predicted comparison group score, it can be assumed that the IRIT treat- 
merit made the difference. 

While' Model C is somewhat more powerful than Model A, its use 
could present a number of problems. As with Model B, the use of Model C 
requires a fairly high level of evaluative expertise and the services of a 
computer Since the special regression design represents a mathematical 
model, the reported results may_.have limited acceptability to an audience 
•that does not understand or trust statistics. . V 

Model C has two additional features which should be considered. 
The design provides for the use of two different but correlated achievement 
tests for pre and. post testing and does not require the administration of 
tests at the usual -norming times/ While testing flexibility and the power of 
the statistical procedure make Model C the second treatment of choice of 




the three models presented, we 
suggest the use . of one rather than 
two achievement tests if only to 
increase the local acceptance of 
the presented data. 
■ y Choosing an Evaluation Model 
Models A, B, and C differ 
in terms of technical rigor and ease 
of implementation. The more rigor- • 
ous the design, the more difficult 
the design may be to implement. 
Model A is a very simple design 

which compares pre to post IRIT treatment gains with test norms using the 
test norm ing group as the comparison group... While it is possible to deter- 
mine differences between IRIT group and norm ing group data, the nature of 
the differences cannot be determined or compensated for/ Since Model A 
compares local IRIT data with national norm data , and not with data from 
other local groups, a call for additional local testing might be expected. 

Model C, the special regression model, is a more powerful design 
than Model A. It compensates for initial test differences between groups' 
using statistical techniques. Since results are dependent on the statistical 
treatment of data, the resulting data may not be accepted at the local level. 



1 , 

Writing an easily understood Model C report-may also present some prob- 
lems. 

Model B, the control group model, is the most rigorous of the three- 
evaluation designs. Its technical merit.depends on the Similarity of the 
treatment and control groups , a .similarity which best occurs- when pupils 
are drawn from a single population and are randomly assigned to the two 
groups. Random sampling is not usually practical 'in_a_^icJschooi^itua z . 
tion. There are, however, some practical ways to deal with this. problem,- 

Using Model B - 
You should now be aware that the control group design, while a 
powerful one, can be difficult to implement in its pure form. The control 
and IRIT treatment groups must be' comparable for the model to work. .Even 
if you were to randomly assign youngsters to treatment a^d control groups, 
' these assignments, would probably go against the grain of the teachers ^ \ 
It was they who identified the students with the greatest needs, and it is 
these same teachers who want those needs to be. met. In all probability, 
your Title I office would also veto this procedure since the most needy 
students must be serviced first. Arid finally, in a school setting it is 
almost impossible to ensure that control and treatment groups are exposed 
to the same educational experiences^ with only one treatment group 
difference... the services of the IRIT. Happily , it may be possible to 
avoid all of these problems. To do this, we should first review some of 



the essentials of the IRIT. , / • 

Your IRIT proposal probably lists three behavioral objectives which 

v i - ■ 

need to be measured. These objectives may be modified or supplemented 

* \ ' & o 

to meet local needs . - „' 

N 

1. IRIT cycle participants will demonstrate reading skill improve- 
merit beydnd that which could be expected without program 
participation. 

2. IRIT pupils will read' at least three books per month or nine 
books per cycle. - ■ 

\. 75% of the IRIT pypils will be able to write simple sentences 
independently at their own level of achievement. * ^\ 

Let's look at the testing plan. Normally, pupils in cycles 1,2 and 
3 of an IRIT program are given 'a standardized achievement test at the 
beginning of their r/fespective cycles. Iri Hartford, the vocabulary ;and ■ 
reading comprehension subtests of the California Achievement Tests 

I 

(CAT-70) are used. At tfte end of each cycle, some 68 to 70 days after i 
pre testing, the pupils are post-tested with the CA3?^70, on the same test 

I 

level with either the same or an alternate test form. ; 

- • ' 1 

In addition to the beginning and end of cycle CAT-70' testing, / 

/ . / 

another set of reading achievement test scbres is needed to avoid thosjb * 

"Model B group selection problems. This /est should include the total ; 

/ • j [ 

population of youngsters who are eligible for the IRIT program. In Hartford 



these scores are obtained from the city-wide spring testing. While Hart- 
ford uses the Metropolitan Achievement Test, 1970 Edition*(MAT-70) , word, 
knowledge and reading comprehension subtest scores from the spring city-^ 
wide testing, other standardized tests may be used. Note theft the city-wide 
test is giv^n as close as possible to the time eff norming. / ' « 

■ Hartford's test plan looks like this. All Hartford youngsters are N 
tested in the spring with the MAT-70, while only\he IRIT students receive 
pre to post cycle CAT-70 testing. 

• . SCHOOL YEAR IRIT TESTING 



CITY-WIDE 
SPRING- 
MAT- 70' 



FALL 



'CAT- 70 
CYCLE 1 
CAT-70 



WINTER 



CAT-70 
CYCLE 2 
CAT-70 



CAT-70 " • 
CYCLE 3 
CAT-70 



CITY-WIDE 
SPRING 
MAT-70 



MAT-70 



MAT-70 
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.Once achievement te^S ere given ? the scores must be converted 
to meaningful information. To do this/ a series ofsteps are followed. , 
Several of these steps^ involve, statistical procedures and can be completed 
more accurately and easily using computer facilities.. The procedures are 
straightforward and are contained in almost any statistical "package " 'so 
that suitable computer services should not be a problem.. Some' suggestions 

for recording the variWs data in a format which can be converted easily to* 

• / ' 

computer punch cards or tapes are contained in, the appendix. 

lfefs review tte> original Hartford VRYi data analyses. The analyses 

meet Model A reporting requirements and are typical of the Title I test score 

/"■ 

data which are available in many districts . Tfiese analyses are also 

' a K . 

required for the use of Model B. To simplify this review, we have repro- ' 
duced an IRIT testing pattern where the MAT-70 is used as. a city-wide 
spring pre-test, while CAT- 70S- are giuen at ,the beginning and end of each 
IRIT cycle. We have also coded ■the i testing times for ease in interpreta- 
tion. For example, CAT-70 testing at the beginning of the first cycle has 
beejn labeled CAT-1; at the end o£ cycle I, CAT-2 at the beginning of cycle 
2, CAT-3, etc. MAT-70 test scores -which have been used as- pre tests 
are shown as MAT-1; while the end of ^ear MAT-70 tests which may be 
used a,s a subsequent cycle's protest have been labeled MAT- 2 . 



MAT-1 



MAT -2 



CYCLE I 




CYCLE 2 



CAT- 3 



CAT- 4 - 



CYCLE 3 




In Hartford the MAT is used as only one factor in the student selec- 
tion process; MAT scores are often used to verify placements which are 
made on the basis of a more comprehensive set of test and performance* 
.data. CAT scores' may be used to remove a[ youngster who is improperly 
placed . This procedure may vary" somewhat from the rigor of the models 
wherein one test is\used to select and another is used' to pre test. From a 
practical standpoint^th'e prdcedures used are reasonable and adequate. for 
use with Models A and B. Model C retired placement on the basis of a 
selection test cut-off score alone. 



Step 1 



Pre tojiost cycle and spring to spring standard scores must be 

1 . _. * i 

m bte 



matched: by cycle, by grade and by subtest, This matching can bte done 
by computer. « - • f 

Step 2 

Comparisons of the» matched pre to post cycle standard, scores and 

' * • I 

the patching spring to spring standard scores can be made using a related 
t-test at the .05 level of confidence. These analyses wil^ help. you t^ 
determine whether,the reported mean change is in all probability a real one 
or 'whether it has come about more likely as. the result of chance'. Remember, 
no evaluation is perfect. Normally, this analysis will yield mean'or average 
scores and standard deviations (SD) . H The standard deviation gives you . 
information regarding the amount of dispersion of the individual scores 
^from the mean so that you can se£""whether the scores have tightened up or 
have tended to move in one or more directions.. If the resultant t-s'core is 
compared with the t-value table found in \/irtually "every statistics book 
and fexcjeeds the .05 level, then in all probability ttfe mean score charige " 
was a real one in 95 cut of 100 cases. Computer services will provide you 

i • 

with ariy v number of analyses on the basis of'the data which are presented; 

by grade, by cycle, by grade within cycle, and for the program as a whole 

I * 

on both the CAT-70 and the MAT-70. f 

" / 

*?ox IRIT programs that do not wish to use Model B, the n6xt step 

I * 

-17- 
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is to interpret and report, the data in accordance with Title I instructions 
and with local ground rules. To meet Title I reporting requirements, pub- 
lishers' manuals should be used to convert mean spring to spring MAT- 70 
test standard scores to percentiles using the norms which were established 
for the spring. The mean percentiles are equated with Normal Curv^Equiv- 
alents (NCEs) and changes in NCEs from spring to spring a\e reported. 

This procedure is spelled out in Step 5. In addition, other ^na.ly^es can 

Si V - 

be reported as needed by your school district or as requested bV^cur state. 

Two*eautions are In order. Models A and B~ require that test data is 
collected at times. whi'ch'ciosely coincide with the fall and/or spring norm- 
ing dates. The models also. suggest that a separate test is used for < • 
student selection and for pre-testing. When the same^test is used for ■ 
student selection and for pretesting as in Hartford' s spring to spring . 
' MAT- 70 test model, an adjustment or regression formula can be used to 
adjust the data so 'as to coincide with fall, to sgring norms. This adjust- 
ment represents a modification to the models / For'Title I projects, the 
adjustment may have to be. approved by your state , ' Sin 9 e the pre to post 
cycle CAT-70 data were not collected at the fall and spring norming points, 
no attempt should be made to convert the standard scores either to percen- 
tiles or to NCEs'. Rather, t-scores, means, and standard deviations may 
be the statistics of choice.- \ 
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* Here we go into Model B*- Do you remember that ModeJ B requires 

i* *■ ■ • . ■ 

a -randomly-selected control group? .For the IRIT program this is not 

i . * 

nec^Ssary, The cycles can act as their qwn controls, * 

Look at -the following diagram. If each of the cycles can be equated , 

with each othef on the basis of a spring MAT- 70 test score using a standard f 

* 'i ■ , <• 

statistical procedure called analysis of variance (AN OVA), and assuming 

that the IRIT will have a decided impact on instruction, c^cle test patterns 

.will look something like this: 

MAT-70 -> 



/ 



MAT-70 



CYCLE 1 



CYCLE 2 




CYCLE 3 



SPRING . ^ •.; * SPRING 

: Notice that each succeeding cycle seems to "start off at a lower 
pfoint than does the preceding cycle. This is supposed to happen if IRIT 
services have provided more growy* than would normally L be expected. When 
this happens, youngsters in one cycle will end up ahead of their peers in 

\ • „ *° , 

' c 

the following cycle who* a re just beginning their IRIT instruction. The key 
to this procedure is to statistically equate the cycles . If the cycles can be 



equated, Model Bean be used. If not, Model A wiil probably have to be 
used . .-.-•« - 

Step 4 v 

Once all cycled have beeVi adjusted, usually by grade level, a 
t-test is.used to determine whether or not the post test of one cycle is 
significantly different from the pre test of the next cycle. CAT-2 is com- 
pared with CAT-,3 , CAT- 4 with CAT-5 , and so "forth under the assumption, 
that if youngsters in all cycles are comparable then the youngsters who . • 
have received the IRIT treatment during cycle 1 should have achieved 
significantly higher test levels at the end of the cycle than their counter- 
parts who are just entering cycle 2. Again mean scores, standard 
deviations, and t-scores can be reported by grade and by cycle. 

Step 5 . 

Now domes the reporting. Using the publisher's test norm conver- 
sion tables , mean MAT-70 standard sco/es are. converted to" percentiles 

using norms which were establishe^for the fall, and/or spring testing 

. .■ j • 1 . \ 

times. Since cycles were not tested with the CAT-70 at two norming 

. points, conversions to percentiles and NCEs should not be attempted. To 

make these conversions, the standardized test must be administered as - 

close to the times of norming as is possible. Mean MAT-70 percentiles. 

are then equated with Normal Curve Equivalents. (NCEs) using a percentile 

NCE conversion table . A copy of this conversion table is contained in the 



appendix. ,NCEs are then reported to the stat$ in accordance with applicabl 
directions and to other audiences in accordance with local ground rules*. 
Step 6 ' , ' ' • 

pnce the mean pre and pos ; t tesrt MCEs have been computed, it is 

necessary to determine, as was done with the statistical t-test, if reported 

• % 

% 

gains.are in all probability' real ones. This is an easy task. Generally . 
speaking/ a gain of 3 or more NCEs is. a salutary one, while 7 or more" , , 
JSTCEs can be considered tq be educafionally important. Tallmadge & Wood 

• ■' ' \ " . r 

(1376) userthe.7 .NCE figure although more recent writings suggest that 
gains, of 3 or more NCEa.can be taken as evidence' that the IRIT program 
has ma<^e a positive impact on the youngsters, and one which is probably 
not due to chance. 

Remember the Model A. assumption th&t the post test expectation 

T V . 

should be the same as the pre test percentile had not the^JRIT treatment 

y 

taken hold? Without getting into deep statistics, some basic assumptions 
about NCEanqJ/or percentile gains are: 

— Wi/thout IRIT treatment, we can expect* no mean percentile or 

N'CE change from pre to p£> states ting, 

/ . . • • ■ * 

- If we get a slight mean percentile or NCE change , this is good 

f~ \& 

j ^nd may be due to the IRIT treatment. 
. - If the mean percentile gain is equivalent to at least three NCEs, 

than the gains are in all probability due to the impact . of the 

■ • * ■ » 
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IRIT treatment. 
- The educational^ importance 
of any gains* should be 
determined at the local . 
^leveli This determination 
should be based oil 'exper- 
•ience with the IRIT program, 
with the tests being used, 
and with- the population be 
being served. 
In addition to the reading 
achievement objective, two other 
objectives are also included in the replication model. These too, must b§_ 
assessed and reported. ' To measure the objective* which states that', 
"Pupils will read atjieast tEree books per month or nine books per cycle", 
compare the number of books read by each pupil witli the established 
standard . - Data can be summarized by number or percentage and reported 
4n graph or- tabular form by -grade, cycle', and for the program as a whole. 
Based on the findings / staff can decide whether or not to adjust the pro-' 
jected numbers of books to.be tfead, place a different emphasis, on this 

k ■ ' , 

^aciing requirement, or adjust the evaluation to- focus on the "why' 1 . Note 
- ' . ' '* * - 

that the coding sheet shown in the appendix contains provisions for 




. • ,. • '--^ \ 

. * - . I . % 

recording the number of books read per student sosthatlthese figures can be 

tabulated or aggregated by computer. j 

■ The; third objectiv<£which slates that "75?!? of, the IRIT pupils will be 

able to write simple sentences independently and a't their own level of 

achievement/' may be somewhat more difficult to measured Whtte formal. 

« 

writing tests have been used in Hartford', the tests have taken, substantial 
amounts Jot time to administer. To eliminate this problem, -teacher judgment 
is used in the evaluation of the ^writing objective!-. * The teacher is dsked to 
indicate at the end of each cycle whether $ youngster can write simple 
sentences independently at, above, or below the. individual's level of 
achievement . These dataware then recorded on the coding sheets. and are 

i 

tallied by computer".*^ This subjective method has several advantages . 

Ratings can pe completed quickly with data obtained from teachers' records 

J \ ' ' ' 

so that me teacher has an input int^ t^e content of the e^luation. Since 

most evaluations are seen as' focusing on the analysis of test data to the 

exclusion of^eacher judgment, this particular rating system is a plus since 

thQ IRIT evaluation is intended to look at program results on the basis of 

several factors^. Good teachers are the most important part of the IRIT 

program and their ratings and recommendations should be considered. 

The Evaluation Report 

■- •. > • . 

Without adequate dissemination ;any evaluation is, virtually useless. 

: s ' ■ .'. : - 

An unread ,and unused evaluation report also represents a substantial waste 



of money and time. If an evaluation is intended to help pedple make 

> 

decisions, then these same decision-makers must be aware of the informa- 
tion which the evaluation report contains. It is important that a systematic 
dissemination plan is developed and that the evaluation is presented in a 
format which will be read. If your IRIT program has been funded with Title 
I monies or is supported by a federal grant, in all probability the funding 
agency will supply reporting forms or will at least suggest a reporting for- 
mat. ' 
*- * . ' 

For Title I programs, state forms are used. While forms may .differ 
s 

by state, all require that certain basic information be reported: number of 
youngsters served, staff ing patterns , dollar allocations by source of 
funding, and objective attainments. 

There is also a requirement to report test scores by grade level, 
test-level, and test form . A copy of the 1980 Connecticut Title I reporting 
form has been Included in the appendix-. While the completion of the 
State Title I reporting form usually satisfies state evaluation requirements, 
the forms are designed to collect and aggregate specific program data. 
They do not contain enough information to facilitate local decisions . ✓ 
Taken alone, the forms t contain minimal information, are difficult for 
parents and lay constituents to read , and leave much unsaid about the 
program. You will need to supplement the required forms with a' reporting 
format of your own. A suggested format has been used in Hartford to 
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provide various constituencies with a reasonable amount of information. 
The format is designed to be easily read and understood. Each year, the 
format is modified slightly on the basis of need. You too, will probably 

want to vary the format or change it to meet local project needs., 

■ c * 
Project Description 

N 

In Hartford, the dissemination plan is based upon reports which are 

i 

written in'plain English and are kept reasonably concise for quick reproduce 
tion and distribution. These reports include enough information for the 
reader to understand what the IRIT project is and how it works. The first 
section of the evaluation report contains two or three paragraphs which 
describe the IRIT program, tells where it is located, indicates the young- 
sters who were served and how many, and shows tuu overall project 

/' 

staffing pattern. In a few sentences the reader is told fxow an IRIT operates, 
y/hy three instructional areas are used, and who is paying for what. If the 
IRIT is specially-funded and a proposal was submitted earlier in the year, 
any operational changes from th^ proposal and the reason why these changes 
were made are also, included. 
Strengths and/or Accomplishments 

The next section of the report describes any program strengths 
and/or accomplishments which may not be picked up as part of the formal 
evaluation.* Evidences of parental involvement such as attendance at 
meetings or comments at open houses , -special student activities, press 
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notices, visitations and visitor comments are included in this section. 
This is the section in which a staff can pat itself on the back, brag about 
accomplishments, and report the kinds of thinjg^that the^staff, parents, 
and youngsters feel really made the differenc^v-^&taff members are encour- 
aged tcTdocument program strengths. This documentation is included in 
the narration. Limitations in documentation are also. reported . It is often 
these unobtrusive evidences which best exemplify program accomplishments 
These should not be neglected. 
Problem Areas • 

« 

In similar fashion, staff members are encoiifaged to report problem 
areas and to make suggestions for change and/or for program modification. 
Documentation is requested. ' 
Evaluation 

This section contains the meat of the evaluation report. While any 
number of formats may be used, in Hartford we: 
1* List the program objectives 

2. Describe the evaluative procedures used > 
a* the overall testing plan 

b. the plan for collecting attitudinal or other kinds of data. 

c. the methods which were to be used to analyze and report 
data 

d. procedures which differ from those which were spe/cified 
in the evaluation plan or in. the proposal; the reasons why 
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chang'es were made are also described 
In sequential order, each objective is listed along withdetailed 
procedures used to measure objective attainment. Here we discuss tests 
used, by form and level, when the tests were given and differences-m-the._ 
testing procedures which were used with different teams, cycles or grade 
levels. If a number of tests or subtests are analyzed and reported in a 
series of tables, this is a good place .to describe the various statistical test 
tests used. * We keep the report simple, but feel that it is important to 
explain what each table heading means, why a standard deviation is 
reported, and why t-tests or other statistics are used. We describe what 
the significance (.05 or .01) or non-significance (NS) table headings mean 
and summarize the findings before the table is inserted.^ , 

* Next, the various data analyses are summarized in tabular form. 3 
Where analyses examine test results on a program, cycle; by cycle, and 6 
grade level basis within each cycle, we describe each analysis and finding 

in one or two sentences for each of the tables. Findings are summarized 

I 

at the end of this section. The summary describes the tabular data and 

indicates whether or not each objective was attained and to what degree. 

* " I 

i 

The procedure is repeated for each of the separate objectives. / 
Evaluative Recommendations ^ 7 

In concluding your report, include evaluative recommendations 

' - I 7 

which arebased on the data which were collected, on the analyses which 

• ' i 

; 'i 
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were performed, and on the findings from the vkious analyses . 
Recommendations should indicate which evaluation areas need tightening * 
up,, amplification, or a different treatment. - * 

Recommendations "hay attend to issues raised by staff but not veri- 
fied by data. The evaluator should make these data limitations clear and 
recommend that the area be scrutinized more closely, during the next project* 
year. An evaluator must be very careful to separate "official" evaluative 
recommendations which are backed up by fact and by data froirv those which 
stem from supposition or .comment. An evaluator might indicate that while 
a staff member made a recommendation it could not be verified as to worth, 
by the evaluator. Since the recommendation seemed to have possible merit, 
it was reported nevertheless; 

Remember our comment that no evaluation is perfect? In order to 

f 

clarify possible misunderstandings while avoiding the possibility ofobvious 
, but unnoticed errors, it is desirable to share the draft evaluation report 
with the IRIT Project Director and staff. While the evaluator has no obliga- 
tion to change a report on the basis of staff input, the prudent evaluated 
knows that the setoff can add any number of practical sugg6stioris to the 
report; they will identify obvious errors, suggest practical corrections, 
and note omissions which if left out will reflect on the evaluator 1 s credi- 
bility. 
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Dissemination „ % 

X ' 
The next step i£ to package the evaluation for dissemination. ; 

Dissemination follows local ground rules. A logical first step is to prepare 
an abstract and share it with appropriate administrators. In Hartford, all 
evaluations, after receiving superintendent approval, are submitted to the 
Board of Education as an agenda item. 

, Pillowing administrative and Board clearance , copies of the evalua- 
tion are furnished to the funding agency, member^ of the IRIT staff, central 
administration, principals of all schools Involved, "and the IRIT Disseminar 
tion Project Eftrector. If^your IRIT is funded with Title I monies^, copies of 
the evaluation should be sent to all Title I schools, all Title^school Parenl 
Advisory Councils, and to the District Parent Advisory Council. Copies of 
the evaluation may be furnished to the local pr^ss and to various informa- 
tion agencies such as ERIC, your State Facilitator network, and the local 
Teacher Center. Copies are made available to the public and to other 
constituencies either by direct distribution or upon request. 

Closing the Loop 
That wasn't too hard was it? And you still have time to sit down 
and have a cup of coffee. Right? Wrong! There ks one more step and that 
is to obtain reader feedback. After you have shared, your evaluation with 
the staff, it is important to obtain* their reactions. Reactions may be 
obtained through open forums or informal discussions. A book could be 
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^written to describe the various methods 
which may be used to obtain feedback. 
Whatever methods $re used, the intent 
is to obtain positive and negative 
reactions as to the content of the 
evaluation and'to the processes used. 
The expectation is that these reactions 
will lead to a better evaluation which 
may help to initiate subsequent pro- 
gram ipiprovemepts . Program *' 
improvement, after all, is the. reason 
for the whole evaluation process . 
'Nuffsaid?' 
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. EVALUATION TIDDLYWINKS 

4 9 

t 

Sin&e no guidebook will cover everything, and since Murphy's Law 
is bouncj to prevail these tiddly winks of evaluation may be of help to the 
project director. 

- Free technical evaluation assistance can be obtained from your 
State Department of Education's Title I office or from the Regional 
Title I funded Technical Assistance Center (TAG) which suppprts 
your state. For further information, addresses, and telephone 
numbers, contact your local or State Title I Director. 

- Title I evaluation models are intended to/cTStess the impact of 
compensatory services using tejst scores which are collected 
close to the time that the normative data were gathered . In 
consequence, the norm referenced models cannot be used to 
assess short term program gains. To avoid this problem with 
IRIT cycles which are operated during the summferieii+her as a 
separate summer school or in conjunction with academic year ' 
operations, three alternatives are suggested 

- For a summer school only IRIT program - Model A should be 
used with'' the youngsters tested in the spring and again in 
the fall with the appropriate ^evel and form of a standardized 
achievement test. Since the spring and fall testing period 
coincide with test norming times, the,data can be reported 
in accord^with Model A instructions . 

- If the summer cycle represents a fourth cycle in a year long 
IRIT program, either Model A or Model B.can be used. Here 
again the cycles will act as controls for each other. To use 
Model B, it is necessary that a spring pre test be used to 
equate all cycles. If your project uses a spring city-wide 
testing, and tests at the beginning and end of each cycle, 
Model B may be appropriate; otherwise, stick to Model A. 

, -"For* a summer program, the norm-reference Model A2 may be 
used with pre and post criterion referenced tests (CRTs). A 
nationally normed test is given in the spring at approximately . 
the same time as the pre- summer CRT 3 . At the end of the 
' summer cycle, IRIT students are again tested with the CRT. 

The median pre-test standard score on the normed test is 
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determined arid the pre-test percentile corresponded to this 
score taken from the test manual. This percentile is the no 
treatment expectation. Using this no treatment expectation 
percentile, a median post test CRT score is- converted to its 
normed test counterpart standard score, which is in .turn • 
converted to a percentile using the post test norms table. 
The derived percentile is the bbservfed post test treatment 
performance indicator. Additional information on this proce- 
dure-can be found in the User's Guide (Tallmadge and* Wood, 
1976). 

of 

While erring may be human and forgiving £ivine, checking ahead 
is even better. If you have any doubts about your state's evalua 
tion requirements, at least give your State Title I Director a call. 
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COMPENSATORY PROJECT EVALUATION REPORTING FORM 



This form is used by local districts in Connecticut to report basic 
skill Title I and State compensatory (SADC) program statistics, to summa- 
rize objective attainments, and to list recommendations based on evaluativ 
) findings. The form^ontaihs sections for reporting test data by grade, by 
test level, and by test form and for calculating standard score and percen- 
tile gains. Note that test score gajfis must be converted to normal curve . 
equivalents (NCEs) to comply with the Title I reporting system. 




r 



• Instructions. 

for "1979-80 Compensatory Project Evaluation Reporting Form" 

PAGET ONE . This page is unchanged fromjast year's reporting form. Note that as 
usual , you are asked to give an unduplicated count of all pupils who took' part in 
-^the program. 
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PAGE TVJO . Except for some slight rewording of the directions at the top of the page, 
.this page is unchanged from that of previous years. 

> 

W.B. TO COMPLETE YOUR EVALUATION SUMMARY, USE EITHER p. 3 OR pp. 4 and 5. * 

PAGE THREE . Use p. 3 to report results of standardized testing (either norm-referenced 
or criterion-referenced) in those projects in which a Title I evaluation model was not 
employed* Also use p. 3 for reporting results of testing among gracfe levels at which 
the models were" not intended, to apply, 

PAGES FOUR AND FIVE- . For those projects and grade levels in which a Title I evaluation 
model was employed, you s ynll reporit^achievement results on* pp. 4 and 5. Please note 
that these pages incorporate much of the information requested previously of "piloting" 
LEAs on a supplementaT reporting form. ' ' ■ \ ~ 

Pages four and five imply a norm-referenced comparison ("A-l" model). If your basic, 
skills project used another evaluation model, ignore these pp. and' provide a summary 
of your "A-2", "B;l" or other model findings. 

Item 1. Instructional Service: Check one box only. Even though this report may be for 
a project which provided both reading and math services (or some other combination), do 
not combine reading and math subtest scores on pp^/4 an/d 5. Use separate pages for re- 
porting the results of reading", language arts and mathematics subtests. A "Basic Skill's" 
project report,' then, may need one, two, or even three such sets (pp. 4 and 5) of achieve- 
ment information. - * ■ 

Item 2 . If norms other than national norms are being presented (p. 5), check the "other" 
box and specify the comparison group: local ,->£.£. 

Item 3. If the same instrument which was used to select pupils for the compensatory pro- 
ject also served as the program pretest, check "yes" to item 3 and include your computatic 
of estimated bias due to "regression toward the mean". v 

Item 4. Please, check the box which best describes the pre/post test interval for which 
you are presenting information. , 

Note: Do not combine achievefnent data gathered over different test" intervals. That is, 
if your project has tested some pupils on a Fall-to'-Spring basis and others Spring-to- 
Spring, you will need separate pages 4 and 5 to report each subgroup's results. 
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Column B: Report h?re the number of pupils by grade level received compensatory 
instruction in the instructional service area checked in Item 1 , Compute column total. 

Column C: Of the number of pupils reported in Column B, for how many are you reporting 
matched pre/post scores? Enter the numbers by grade level and compute the* column total. ' 

The rest of the information on p. 4 is identical with information requested on p. 3 of 
previous years' reports. If you are reporting results from instruments which have raw 
score-to-percentile conversion tables (rather than scaled score-to-percentile conver- 
sions), please strike out "x.S.S." in the last column, p^4, and indicate that you are^ / 
presenting mean raw scores instead. 

Column D: Report here the percentiles associated wUh the mean pre-and posttest scores 
reported on page four. Caution: Do not average the percentile ranks of individual pupils. 

ColumnE: Convert the pretest and posttest percentiles -in column D to their associated 
normal curve equivalents. Alternately, mean N.C.E. 1 s may be reported. 

Column F: Post N.C.E. minus pre N.C.E. = N;C.E. Gain. (Losses are reported as negative 
numbers, of course.)^' 

« 

Column G: Compute a Weighted N.C.E. Gain by multiplying the observed change (col. F) 
by the number of pupi 1 s pre/posttested (col. C, p. 4). Compute the. column sum. 

To •Complete page 5, divide the column G* total by the column C total. This is the project's 
weiohted mean gain for this particular instructional service area. . . 
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1979-80 COMPENSATORY PROJECT EVALUATION REPORTING FORM 



Schotil District 



District Address' 
Project Title 
Director 



Evaluator 



(Name) 



Program Site (a) 



, (Telephone) 
'(Telephone) 



(1-3) 



(5-6) 



(8-9) 



(11-12) 



PLEASE SUBMIT 
JHQ COPIES 

THIS REPORT IS DUE 6/20/80 

C.S.D.E. 

State Office Bldg., Ri. 375 
P.O. Box 2219 

Hartford, Connecticut 06115 



Funds supporting this component: 

■Title I: 

SADC public: , 

SADC non-public: 

' Other (specify): ■ 
TOTAL: 



.(14-20) 
.(22-27) 
.(29-34) 
(36-42) 



Expenditures included in Total above which 

supported services to private school 

children: (44-49)/ 



Unduplicated count of program participants by grade levels: 



PK 


K 


, 1 


' 2 


■ 3 


4 


5 


6 


7 


8 


9 ' 


10 


11 


12 


TOTALS 


(8-10) 


(12-14) 


(16-18) 


(20-22) 


(24-26> 


(28-30) 


i 

(32-34) 


(36-38) 


(40-42) 


(44-46) 


(48-50) 


(52-54) 


(56-58) 


(60-62)1 






. ' PRIVA 


TE SCHOOL' 






X 


(12-14) 


(16-18) 


(20-22.) 


ML 


(28-30) 


(32-34) 


(36-38) 


(40-42) 


(44-46) 


(48-50) 


(52-54) 


(56-58) 


am 





Number and full-time equivalent of project staff paid by compensatory funds: 
Instructional Other Professional 



No. f.t.e. 



Teachers 



/n i ^ \ No. f.t.e. 
(Specify) — 



Clerical or Other 

re * ic \ No. f.t.e. 
(Specify) — 



(8-9) (11-15) 



Aides 



(18-19) (21-25) 



School Diatrict Component Title , / Pa 8 e 2 

'Using this page, (1) State the performance objectives^ for this component (from the Application); (2) Specify the 
measure(s) u$ed to evaluate each objective; (3) Indicate the method of analysis applied to the data collected with 
each instrument; (4) Present the results of the evaluation. At the foot of the page state one or-more program 

recommendations based on the evaluation findings.,/ 

i/ 
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PERFORMANCE OBJECTIVES 


DATES 

INSTRUMENTS/ADMINISTERED 


TKEATMENT OF, MA 
INCLUDING TESTS 
M SIGNIFICANCE 


RESULTS 


i * 

* 




• 


• 

r 1 


1 t— y 



RECOMMENDATIONS: 



School District 



Component Title 



Page 3 



FOR PROJECTS/GRADE LEVELS WHICH ARE N£ USING A JITLE l" "BA^C SKILLS EVALUATION MODEL - 

Use this page to report any standardized test information' which was collected and analyzed as part of the evaluation desigr 
of any project which wafnot designed primarily to increase pupils' achievement in reading, mathematics or language' arts. 
Use this page, too, to report standardized test information for groups of pupils below grade level two, even though these 
pupils may have participated. in a Basic Skills Project which also served older students. 

In structions : Present scaled score averages by grade level for each test/subtest analyzed, for instruments which provide 
raw score-to-gercentile conversions, present raw score ayeraaes instead. In this case, please cruss out the words "Scaled 
Scores" and label these columns as "Raw Scores". Recording of percentiles on this.paqe is optional. If you frovide par- 
. centiles associated with average scores, please indicate the type of norms employed. " * , 

: 0 MtlOMl WMW 

[j other noras 
{specify) 
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XCl 











































Jest Instrument Inforwtion 



Scaled 

Scosss 



Name 

of 

iybtest 



Pre/Pos* I Month of 
Battery I Prt/M Pre/Post 
' Me) I Form 'tests 



Test 
I 

S.S. 


Post 
Test I 

S.S. 
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(OPTIONAL) 
Percentiles Associated ' 
With Average Scores; , 
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" f 1 1 77 
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School District 



Component Title, 



' ■ (1-3) ' (4-8) 
FOR BASIC SKILLS PROJECTS WHICH SERVED PUPILS AT OR ABOVE GRADE TWO, USE THIS PAGE 



(9) 



1, INSTRUCTIOfjAL SERVICE (Check One) Q READING Q LANGUAGE ARTS Q HATH 4, TESTING PATTERN 



2. TYPE OF NORHS USED IN SCORE CONVERSIONS Q NATIONAL Q OTHER 
fior ' • ■ PpecffyT- 

V3. HAS THE PRETEST INSTRUMENT USED TO SELECT PROGRAM PARTICIPANTS? Q NO Q YE5n, Q More than 12 months 
. {IT) Column 



Fall to Spring (12) 
Spring to Spring 
Fall to Fall 



Grade 
Level 


No, of 
Pupils Who 
Received 
Instruc- 
tional ' 
Servir.p<; 


NQ.Of 

Pupils 

Pre^Post 

tested 




r 






A 
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\ 












■ 




































t 

i 





(|3-14) (15-16) (17-18) 



r Name of 
. ! Test .' 


1 . ' 1 

Edition 


f 

f 

Name' of 
Subtest 


Pre/Post 
Battery 
Level 


■ 

Pre/Post 
Form 


Month 
oft 
Pre/Posl 
Tests 


Pre 

Test 

7 

s,s. 


1 

Post- 
Test 
x * 
S.S, 
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d , Total TotaTPre/ 
ERiGO Served Post Tested 



School District 



Component Title* 
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(continued from p. 4] FOR BASIC SKILLS PROJECTS WHICH SERVED' PUPILS AT OR ABOVE GRA[fe TWO 

' Weighted mean project gain; 



Column D 



Column E . 



Associated • 
• Percentiles 


Pre ' 


• Post 






Hi 












! 
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v ■ Associated 
N.C.E.s, , 
(orxIMi.) 


Pre 
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ERLC 



,48 



(21-22) 



'ToU, Column G 



YoU, Column C . 
Column F, Column G 



H.C.E. ,> 
Gain 



L Reading 
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PROJECT CODING SHEET 



To gather compatible evaluation and eligibility data, 8-1/2" x 17" 
project data coding sheets are used by each of Hartford's basic skill 
compensatory programs. Since many, of these programs are also funded 
with State compensatory monies, free milk/lunch eligibility data is 
collected. Thus, a general Evaluation/Eligibility Qoding Sheet is used to 
record eligibility .and demographic data as well as city-wide spring-to- 
spring (MAT-70) test scores which are 'collected as part of a common eval- 
uation model. 

■ ; 

A second coding sheet, which differs from project to project, is 
used to record IRIT CAT-70 standard scores (ADSS), numbers of books read, 
and the level at which students 'can write simple sentences. Provisions 
are included for, recording the number 'of days pjreserit and the number of 
days enrolled in the IRIT program . These figures provide, a basis for com- 
puting Percentages of attendance and the number of cycles in which a 
youngster has been enrolled in the IRIT program during the current year.. A 
sequential student number (Column 1-4) is used to merge data after these 
have been punched into two 80-cblumn cards. 
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EVALUATION/ELIGIBILITY CODING SHEET INSTRUCTIONS . 

Edition: 1979-80 

CARD 1 • 
Vari able 

--Number students sequentially; 1 - 

School Year: use last digit (1979-80 =0 etc J 

Name: Last Name, First Name, Middle Initial 
Space between last name, first name, middle initial / 

* Student Identification Number: assigned by school^ 

.Eligibility for: 

f Free Milk . = M 
Free Lunch = L 

. School Attendance Area Code: See instructions 

GricPe: ' -\ • 

MAT Placement Tests: use ^receding spring test scores 
only. ^ - ^ 

MAT Placement Level: Elementary:. = E, Intermediates =. I, 
Primary 1 =P1, etc. " : 

MAT Placement Test Form 

'Placement: MAT Word Knowledge (SS) 
Placement: MAT Word Analysis (SS) 
Placement: MAT Reading' (SS) 

Placement: MAT Math Computation (SS) 
Placement: MAT Math Concepts (SS) 
Placement: MAT Math Problem Solving (SS) 

MAT Post Test^ Level 
MAT Post Test Form 



Column Variable 

58 - 60 Post Test: MAT Word Knowledge (SS) 

61 - 6.3 Post Test: MAT Word Analysis (SS) 

64 - 66 Post Test: MAT Reading (SS) 

67 " 69 fost Test: MAT Math Computation (SS) 

7.0 - 72 Post Test: MAT Math Concepts (SS) 

73-75 Post Test: MAT Math Problem Solving (SS) 

77 Center Code: 

1 = Clark 

2 = King 

3 = Wish 

4 = Kinsella 

I 

78 - 79 Program Code: See instructions 

• 80 Card Number: 2 



Evaluation Office 

1979-1980 

EE-IPJT 



INTENSIVE READING INSTRUCTIONAL TEAMS Edition: 1 

CARD 2 

' ■ GoluI I},C V ariaUle 

Sequential Student Number - Same as Card 1 . 
Student Initials - Same as Card 1. 
'School Attendance Area Code - Same as Card 1. 



1-4 
5-7 
8 - S 



10-11 Grade - Same as Card 1. 

12 Center Code: 

1 = Clark 

2 = King, - . . 

3 = Wish ^_ 

4 = Kinseila 

13 Cycle: 1 , 2 , or 3 
14-15 CAT Pre Test Leve'l 

!6 CAT Pre Test Form 

17-19 Pre Test: CAT Vocabulary (ADSS) 

20 " 22 Pre Test: CAT Comprehension (ADSS) 

23 _ 25 Pre Test: CAT Total Reading (ADSS) 

26.-27 CAT Pet Test Level 

28 CAT Post Test Form 

29 ~ 31 Post Test: CAT Vocabulary (ADSS) 

" 2 ~ 3 4 Post Test: CAT Comprehension (ADSS) 

35 " 37 Post Test: CAT Total Reading (ADSS) 

\ 

3 8 - 40 Number of books' read 

42 Student can write simple sentences: 

.above instructional level = 1 
at instructional level = 2 
below instructional level - 3 



Column 
43 - 45 
46 - 43 

49 

80 



Variable 

Days Present 
Days Enrolled 

Cycles enrolled in IRTT this year (count present cycle) 

c 

Card Number: 2 



A| ; 


\ MODEL 

\ B| 
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i \ 


'' MODEL £ 
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IRIT REPLICATION REPORTING FORM 



This form, which is based on Connecticut's State Title I reporting 
form, can be used for reporting replication site evaluation data to the 
Hartford Demonstrator/Developer (D/D) project in accord with the attached 
instructions . 



IRIT REPLICATION REPORTING FORM 

Please send a copy of the attached reporting form to the D/D project 
director at the end of each school year. Information from all D/D projects 
will be compiled with test data analyzed by grade, by site, and on an over- 
all program basis* This will be done not only to help the funding agency 
determine the extent to which the overall IRIT replication process is working 
but to feed back this information to each project director as well. Since the 
reporting form is based on a Connecticut Title I form, if your IRIT has received 
Title I funding, in all probability this information will already be available. 
Hence, it should be a simple matter to transfer this information on to the 
attached forms. Specific questions regarding this reporting should be 
directed to the D/D coordinator at (203) 566-6627 or to the Hartford Public 
Schools' evaluation office by calling (203) 566-6074. 

•. PAGES 1 and 2. Self-explanatory. 

* PAGES 3 and 4. Title I evaluation model, and other achievement results* 

results should be entered in the appropriate spaces. 
Note that pages 3 and 4 were constructed for use 
with a norm-referenced comparison (A) model. If / 
your IRIT project used another evaluation model, 
ignore these pages and provide a summary of Model 
B, C, or other findings. 

•. ixem 1, If norms other than national norms are being presented 

^ (p. 4), check "other" and specify the comparison 

group: local, State, etc. 

ITEM 2 . If the same instrument which was used to select or 

confirm pupils for the compensatory project also, 
served as the program pre test, check 'Ves" and 
include your computation of the estimated bias due 
to regression toward the mean. 

ITEM 3. Please check the box which best describes the pre/ 

~ post test intervals for which you are presenting 

information. Do not combine achievement data 
gathered over different test intervals. That is, if 
your IRIT tested some pupils on a fall-to-spring 
basis, and others on a spring-to-spring basis, use 
separate pages 3 and 4 to report each sub-group 
results * 



56 
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MNS B AND C. 



Note that^Column B" should contain the total number 
of pupils who received IRIT services by grade, 
while Column C should show the' number of these 
youngsters for which matched pre/post test scores 
were available. 



COLUMN D. 



Report here the percentiles associated with the mean, 
pre and post test scores reported on page four! 
Caution: Do not average the percentile ranks of 
individual pupils. 



COLUMN Eo • Convert the pre test and post test percentiles in 

column D to their associated normal curve equiva- 
, lents. Alternately, mean NCE's may be reported. 

Post NCE minus pre NCE = NCE Gain. (Losses are 
reported as negatj^e numbers , of course.) 

Comput^ a Weighted NCE Gain' by multiplying the 
1 observed change (column F) by the number of pupils 

pre/post tested (column C, p. 3). Compute tlje 
column sum . , • . 

h \ ' 

To complete page 4, divide the column G^otal by the column C total. Tfiis 
is the project's weighted mean gain for this particular instructional service 
area. N 



COLUMN F, 



COLUMN G. 



/ 



IRIT REPLICATION REPORTING FORM 



School District _ 
District Address 
Director 



(Name) 



Evaluate 



(Name) 



(Telephone) 



(Telephone) 



Funds supporting this component: 
Other (specify): 



Program Site(s) 



Muplicated count of program participants by grade levels 

PUBLIC SCHOOL 



Y 



2 


3 




5 


6 


7 


8 


,. 9 


. .10 


11 


12 ' 


■ TOTALS 












PRIVATE 


SCHOOL 
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• 3 
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r 
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8 
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•-10 
L 


11 


-12 


TOTALS 



Number and Ml-tke equivalent (f.t.e.) of project /staff: 

Instructional ' 
No^ f.t.e. 

6o 



Clerical or Other 

* "o. _._ f.t.e.. 



Specify 



Teachers 
Aides 
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i'h(Y>1 Patriot 



UeLt this page, (1) State the performance objectives; (2) Specify the measure(s) used to evaluate each objective; 
(3) Indicate jhe rettoi nf analysis applied to the data collected with each instrument; W) Present the results of 

'^.H "<n 



PERFORMANCE OBJECTIVES 



DATES 

INSTRUMENTS / ADMINISTERED 



TREATMENT OF DATA 
' INCLUDING TESTS 
jfflR STCHTTTflAHCg 



RESULTS 



RECOMMENDATIONS , FOR PROGRAM MODULATION OR ITfROVTOT: 
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Continued* 
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School District 



(continued from p. 4) FOR BASIC SKILLS PROJECTS WHICH SERVED PUPILS AT OR ABOVE GRADE TWO 

Weighted mean project gain: 
ToVoi, Column G 



Column D 



x 
x 



Associated 
Percentiles 
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Post 
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PERCENTILE TO IMC'i.; CONVERSION TAB] E 



This table, which was derived from the table of areas under the nornuil 
curve found in most statistics books, gives the NCE that corresponds to each 
percentile (all numbers are express d to 3 significant figures). By using ^tsjijs 
table, observed and expected posttest percentiles can be converted directly to 
NCEs. The NCE gain is the difference between the observed NCE and the 
expected NCE. 
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STANDARD DEVIATION UNITS 



2 5 



Area under the normal curve divided into NCEs, 
percentiles, stanines, and standard deviation units. 
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INTENSIVE READING INSTRUCTIONAL TEAMS ' 

The focus of the Intensive Heading Instructional Teams is remediating 
skill deficits and provfd lng\ students with the opportunity to apply and 
practice their reading skills, A student completing the cycle should 
function r;ore effectively when returned to the classroom reading program. 

The directed ^eading activity is a part of the I.R.l.T. Instructional 

* *" 

format but is not the primary focus. Our intent is to provide daily inten- 
sive individual and small group skill instruction jwhich is usually difficult 
to facilitate in a regular classroom of twenty- five students or more. # 

To effectively remediate it is usually necessary to use materials at 
a student's independent reading level. A student will then focus, on the 
r.kill snd not be confronted with unknown vocabulary or concepts. 

Special cmph3F<s is placed on facilitating communication through writing. 
V/<^ v:ant students to express themselves w5th ease and clarity. Good spelling 
ability Is on" tool in this process. The spelling lists in the Hartford 
Language Ari.b Curriculum Guide arc the high utility words practiced and 
used in ' jv/ written activities. 

Ch^ren are encouraged to. use their best penmanship when doing written 
work, an the need arises, problem letter formation fdr individuaj/chi Sdren 
is 'r.'/isd**'. -J. The 7-:/ v: ; i-vr/cr method Us use>i to assure continuity with 
rJassrocm Instruction. 

In the area of enrichment and individualized reading, students have the 
opportunity to apply their reading skills in books of tJiesr choice, Reading 
daily gives the practice that nurtures. better readers, stimulates wri ting and 
develops vocabulary. * * 

Direct vocabulary Instruction m }i' Incorporated In the three instructional 
areas. The basal reader vocabulary »* stressed es wdll as high utility 
words. » . j" 

7<i ' , . x 



CRITERIA FOR SELEC TION OF STUDENTS FOR IRIT 



Children should be recommended vho are below grade level in reading 
achievement. 0 

Children must be able to work successfully within an intensive program and 
adapt to the organizational set-up. If children are retired who cannot 
adjust to the situation, it may be necessary to exclude them from the 
program. 

Because of the intensive nature of the program, no new students or replace- 
ments will be accepted after the first two weeks of the cycle. 

No pupil should be recommended who is now attending another special program, 
except under certain circumstances when the teacher and IIJIT feel the. student 
would benefit from the program. 

The benefits to a school are increased when there are no more than four or 
five teachers involved in each cycle of the program* 

Experience has indicated that preference should be given to. students who have 
a good attendance record. However, other contributing factors will be 
considered. 

Students who have a high degree of unexcused absences i^Lll be dropped from 
the program. 

Guidelines to lae used for selection of students should include information 
found in the cumulative folders, teacher evaluations, principal, parents and 
reading consultant recommeiarclations. 

Although the teachers are requested to recommend for consideration as many 
children as they feel would benefit from this type of instruction, it must 
be clearly understood that not all uf the recommended children can.be 
accepted in this pro-ram at any one time. 

Students must have scored below the 23%ile on the M.A.T. or another stand- 
ardized test. Students with higher percentile scores may be accepted if^ 
the need is documented through other testing. 




10. Students in Bilingual Classrooms are eligible if they meet the above 
cations and have begun receiving instruction in reading in English in 
classroom. Please note that students should not have begu£ English r^d)Lng 
instruction. in the classroom unless they have attained a' 2 reading level in 
Spanish and are at least in level 3 of ESL for oral English proficiency. 
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PLEASE ENTER AT THE CLOSE OF EACH CYCLE 

I.R.f.T. P.M. ACTIVITY LOG 



r 



t. PARENT CONFERENCES 
Cycle 1 ____ 

Cycle 2' 

Cycle 3 ' 



PARENT. WORKSHOPS/ IN SERVICE 

Cycle 1 

Cycle 2 

Cycle 3 ' 



3. ' CONFERENCING WITH CLASSROOM TEACHERS. 



Individual ly 



Small Group 



Workshop/ In Service 



Cycle 1 
Cycle 2 
Cycle 3 
Cycle I 
Cycle 2 
Cycle 3 
Cycle 1 
Cycle 2 
Cycle 3 



k. PREPARATION OF MATERIALS FOR CLASSROOM TEACHERS 

Cycle I • 

Cycle 2 

Cycle 3 



5. ATTENDED S;S.T./P.A.T. 



Cycle 1 
Cycle 2 
Cycle 3 



6. OTHER SERVICES 



MSW:, rk 



7 ii 



1983 



NAME 



GRADE 



TEACHER 



HOKE ADDRESS_____ 

PARENT OR GUARDIAN'S NAME_ 
EMERGENCY INFORMATION - 

— ' Name 



ATTENDANCE RECORD^ 

LANGUAGE SPOKEN 'IN HOME 
SPECIAL HEALTH PROBLEMS 



ID? 



Program Exit Date 
SCHOOL 



ROOM BIRTH__ 

SEX PHONE' 



(Business Phone), 



"Address 
Attended IRIT_ 



Phone 



(Dates) 
(Center) 



Other Remediation Services • \ ^ 

(Dates) 



TESTS 



METROPOLITAN ACHIEVEMENT TEST 



Date 



(tree, sprlny oulyf 



I acemcnt 



TeveT 



Reading 
SSi 
S. 
%'. 

Other Tests 



Language « 

S. 

%: 



"Test Form 



Math 

sTT" 



Last Book Completed^ 



Date 



CLASSROOM TEACHER'S COMMENTS: 



7 a 



, l.R.l.T, READING RECORDS 

I.R.I .L CENTER __Cycle School 

IPrelDate-upper box (Post) Date-lowe 

■a 

Metropolitan Survey ^Stanforp" Diag. Reading 



STUDES! ID? 
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ESCUELAS PUBLICAS DE HARTFORD 
Hartford , Connecticut 



Estimados padres: 

Su rftfio/nina esta' asistiendo a uno de nuestros programas de 
Lectura Intensive. Nosotros quisieramos saber en que forma 
nuestro prograv : c* beneficiando a su nino/nina. 

Favor de marca^ vv». fx) en las siguientes preguntas: 



!• <iLe gusto' a su nifio/nina el programa 

I.R.I.T.? si 

2. <<Que' le grusto a su nifio/nifia del 
programa I.R.I.T.? 



3. <jLe desagradrf algo del programa I.R.I.T. 

a su nifio/nifia? Si 

4. d'Lee mas su nino/nina en su casa? . si 



Ho 



No_ 

No- 



No 



5. dEsta' su nino/nifia mas interesado en 

la escuela ahora? Si No 

6. CVisito' usted el .programa I.R.I.T.? si 
■ 

7. dTiena us ted algo que sugerir paxa mej orar 

el programa I.R.I.T.? 



Nombre del estudiante 
Ncrnbre del Padire 



Escuela Fecha 



Por favor devuelva esta forma al Centro de Lectura lo mas 
pronto posible. 



I.R.I.T. 1981-82 
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HARTFORD, PUBLIC SCHOOLS 
Hartford , Connect i cut 



Parent; 

Your chil,: is presently attending one of our Intensive Reading Instructional 
learns. We would like to know how this program is benefiting your child. 

Please check (x) the questions listed below': , 
"i. Did your child enjoy attending the I.R*. I.T.? Yes No 



Wh t did your chi* ' like about the I.R.I.T.? 

3. Was there anything your child d ; sliked about the I.R.I.T.? Yes Mo 

Does your child read more, at home now? Yes No 
5. Is your child more interested in school now? Yes No 
•6, Did you visit t^e I.R.I.T.? Yes No 

7, Do you have suggestions for improving the I.R.i.T. Program? 



Chi Id 1 s Name Parent's Name 

School Date 



Please return this form to the Reading Center 



1 .R.I .T. . 1981-82 
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I.R.l.T. STUDENT EVALUATION 



__DO NOT USE 

1.. Check one . Boy Girl 



2 - Check one Grade: 3^5 



3. Did you like going from class to class? Yes Uo 
Why? 



4. Did you like having three teachers? Yes No 

5. Has I.R.l.T. helped your reading? Yes No 



How? 



6. I 



s your family happy with the work you did at I.R.l.T.? 
How? 



7. Do you want to. go to reading school again? Yes No 

Why? 



8. Write one thing that you would tell a child about 
I.R.l.T. 



i 



I.R.l.T. 1981-82 
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Hartford Public Schools 



Administrative Offices 

249 Hioh Sireet 
Hertford, Connecticut 06103 



TO: CLASSROOM, TEACHERS 

SUBJECT: INTENSIVE READING TEAM EVALUATION 

A numbcr'of your students have been'attend ing one of our Intensive Readin 
Teams. In'order to report on the program to the State Department and to 
improve our services to you, we would like you to complete the enclosed- 
evaluation form. 

We would appreciate it very much i f you would return the form. 
Thank you for your cooperation in this matter. 



,MSW: rk 
Enc. 



HARTFORD PUBLIC SCHOOLS 
Hartford, Connecticut 

I.R.l.T. PROGRAM 
Teacher Evaluation Form 

Name Ghade School 

Date _____ • 

The number oT your pupils who attended the I.R.l.T. Program this cycleN 

Please answer the following questions in relation to the pupil's standing 
in your classroom at. the beginning of this cycle. 4 

1. Have you noticed the improvement in skills of the children attending t! 
I.R.l.T.? Yes No . Describe briefly.- 



2. Have your children developed a better attitude towar.d read i ng? 
Yes No . 

3.. Are your children reading more during their free time? 
Yes No 

A. Have you noticed an improvement in students' work, habits? 
Yes No • ' 

5. a)Wcre you able to attend the I.R.l.T. Open House? Yes No 

b)Wcre you able to visit the I.R.l.T. Program at any other. time? 
m *Yes No . 

6. How many pupils remained with you during the A.M.? 



7. , What effect did removing some. pupils from your classroom have on your 
program? 



8. Do you have any. suggestions to improve the - s l .R. I .T.? Yes No 

What "are tfiey? * 



I.R.l.T. 1981-82 
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ANNIE FISHER SCHOOL - 
HARTFORD, CONN. 



